The unreasonable effectiveness of BM25 for agentic search

GPT-5 is shockingly good at search, and that changes the “BM25 as a baseline” story. Using GPT-5 search trajectories from BrowseComp-Plus, I’ll show how default BM25 parameters and evaluation harnesses can make lexical retrieval look weak, while real agent queries often play directly to BM25’s strengths. Much like grep became a core retrieval primitive for coding agents, BM25 is re-emerging as a powerful primitive for agentic search.

Jo Kristian Bergum

Hornet.dev

Jo Kristian Bergum is the CEO of HORNET.dev and a 25-year veteran of the search industry, formerly serving as the Chief Scientist at Vespa.ai and a Distinguished Engineer at Yahoo.